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Abstract: Throughout the United States, state legislative bodies, university governing boards, and regional and professional 

accreditation organizations are increasingly requiring universities and colleges to demonstrate that they have appropriate 
self-regulating processes in place to achieve their stated missions and goals. Central to these processes is that they be based 
on outcomes assessment and be focused on continuous improvement. This paper presents an overview of Georgia Tech's 
experiences in designing and developing outcomes-based assessment programs for all its undergraduate and graduate programs 
with an emphasis on those experiences within the College of Engineering. It also presents the current efforts to implement the 
programs within a continuous improvement process that is intended to be sustainable and broadly responsive to the needs of 
multiple external constituents. It concludes with seven suggestions for those embarking on developing outcomes-based 
assessment programs. 
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1 Introduction 



Universities and colleges throughout the United States are increasingly being required by external governing bodies and 
accreditation organizations to demonstrate that they have appropriate self-regulating processes in place to assure that they 
are achieving their stated missions and goals [6][7][8]. A central element required of these self-regulating processes is that 
they be based on outcomes assessment and be focused on continuous improvement [2][9]. The new Engineering Criteria 2000 
(EC 2000) issued by the Accreditation Board for Engineering and Technology (ABET), the accrediting organization for U.S. 
engineering education and engineering technology programs, is an example. These criteria are dramatically altering the manner in 
which engineering colleges evaluate their curricula, interact with their major stakeholders, and seek accreditation [2][3][4]. 
These criteria, and similar expectations by other external constituents, are also placing additional demands on limited university 
resources and faculty time to respond to these new requirements [5]. At the Georgia Institute of Technology, all degree 
programs, both undergraduate and graduate degrees, now have outcomes-based assessment programs. They have been 
developed over the past several years either as the result of a mandate by the Board of Regents, the governing board of the 
University System of Georgia, or to meet the new professional accreditation standards, such as those by ABET, the Computing 
Sciences Accreditation Board (CSAB), the National Architectural Accrediting Board (NAAB), and the American Assembly of 
Collegiate Schools of Business (AACSB), or to fulfill the requirements for accreditation by the Southern Association for Colleges 
and Schools (SACS), the accrediting body for secondary and post-secondary institutions located in the southeastern region of 
the United States. Maintaining these potentially resource-intensive programs has become a considerable challenge. The challenge 
is to implement sustainable outcomes assessment programs that facilitate the Institute’s efforts to improve its educational 
programs while also demonstrating to multiple external constituents that these programs are effective means of 
self-regulation. 

This paper describes some of Georgia Tech’s experiences in designing and developing outcomes-based assessment programs and 
how it is now implementing them in a process intended to be both sustainable and broadly responsive to the needs of multiple 
external constituents. An emphasis of the paper is on the experiences and activities as they relate to the College of Engineering. 






ft 



The paper first presents a brief history of Georgia Tech's activities in the area of outcomes assessment. Because outcomes 
assessment has been driven by both internal initiatives and external requirements, the Institute now finds itself, somewhat 
predictably, at the juncture of consolidating and refining the results of these activities into an overall effort that is affordable 
and effective. Thus, the subsequent two sections summarize the current state of affairs. The first of these two sections 
presents what the College has concluded are the basic elements needed for an effective outcomes assessment plan. While there 
clearly are many elements which may be included in an outcomes assessment plan, the College has identified four elements which 
have been found to be most helpful for its internal needs and that are common to the needs of its principal external 
constituents. The process to implement these plans is then presented. The process is designed to provide adequate 
documentation to satisfy both internal needs for self-evaluation and external requirements for self-regulation without requiring 
a significant administrative effort. Finally, the paper concludes with seven suggestions for those embarking on developing 
outcomes-based assessment programs. 
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2 Evolution of outcomes-based assessment at Georgia Tech 

The creation of outcomes assessment programs at Georgia Tech evolved during most of the 1990s. The foundations for these 
programs were laid largely during the first half of the 1990s through a number of internally-initiated events focused on 
continuous quality improvement. The translation of some of these activities into formal outcomes-based assessment programs, 
however, occurred largely during the later half of the 1990s as a result of the need to meet the requirements of two 
re-accreditation reviews. 

In 1992, three events laid the foundations for the first College of Engineering and Institute efforts in outcomes assessment. The 
first event was the development of a College of Engineering Strategic Plan in conjunction with the development of strategic plans 
for each School within the College. An important part of the strategic planning process was the identification of sets of 
programs at peer institutions from which the College and Schools would benchmark. The College identified initially twelve peer 
colleges, and each School identified its own set of peer schools or departments. These sets have been refined over time as a 
result of surveys of Georgia Tech undergraduate and graduate students, alumni, and employers of Georgia Tech graduates. 
Considerable information has been gathered regularly from these programs. The second event was an award from IBM whose 
purpose was to facilitate the implementation of Total Quality practices across Georgia Tech. While the grant affected the 
administrative processes of Georgia Tech more than it did the academic processes, it nonetheless helped move the Institute 
further down the path toward better continuous improvement processes. Finally, the third event was a mandate from the Board 
of Regents of the University System of Georgia that all undergraduate programs within the university system would develop and 
implement outcomes-based assessment programs over a multi-year period [8]. This mandate initiated Georgia Tech's first 
Institute-wide effort in outcomes assessment. 

By the mid-1990s, several other activities continued to build off these early efforts and lend more focus to outcomes 
assessment. The College of Engineering formed an Ad Hoc Engineering Curriculum Assessment Committee comprised of 
representatives from several companies that were known for their excellence in continuous quality improvement. Also included 
on the committee were faculty, staff, and students from the Institute. The purpose of the committee was to learn and adapt 
from industry those continuous improvement practices that might be applicable to academe. Among the principal results from 
this activity was the identification of the College of Engineering's key stakeholders from which the College now seeks feedback 
on a more regular and systematic basis. A major update of the Institute Strategic Plan was performed in 1995, and in 1996, the 
Board of Regents mandated the conversion of the University System from the quarter system to a semester system by the fall 
of 1998 (although Georgia Tech was given an extension until the fall of 1999 because of the disruptions associated with its 
serving as the host of the 1996 Olympic Village). Needless-to-say, much of the previous strategic planning, benchmarking, and 
efforts in continuous improvement and outcomes assessment were helpful in planning and designing the new curricula. 

During the late 1990s, several events coalesced these activities and resulted in formal, published outcomes assessment 
programs. Principal among them for the College of Engineering were two re-accreditation reviews. One review occurred in 1997 
when Georgia Tech served as one of the first five institutions to pilot ABET's new accreditation criteria, EC 2000. Ten 
engineering baccalaureate programs and one masters program were reviewed [2][7]. The other review occurred in 1998 and 
involved eighty graduate programs across the Institute as part of a special re-accreditation visit by SACS [6]. Because 
outcomes assessment is now a part of all the Institute's undergraduate and graduate programs, the Institute recently hired a 
full-time Director of Assessment to provide guidance and counsel on improving the individual unit plans and to facilitate more 
cohesion among them across the Institute. Within the College of Engineering, this is aided by support from SUCCEED, an 
engineering education coalition sponsored by the National Science Foundation, which targets outcomes assessment as one of its 
four major areas of activity [10]. 

Georgia Tech's experience in designing and developing outcomes-based assessment programs was anything but straightforward. 

It required the involvement of a large number of faculty, and many, many meetings. However, a guiding philosophy throughout 
their development was to create programs that would work within the existing resource allocations, utilize well assessment 
activities that were already in place, provide a good baseline of initial results, and, of course, meet the external needs required, 
e.g., EC 2000, SACS. Although each effort attempted to build on the preceding activities and to capitalize on the growing list of 
"lessons learned", a considerable amount of time and resources were required for each effort because of the expanding list of 
internal and external constituents involved and an expanding list of needs and requirements [2][3]. 

The principal challenge now is to implement the programs so that they are both broadly responsive and sustainable; specifically, 
they need to provide useful information for multiple constituents and within a framework of limited resources and faculty time. 
Within the College of Engineering, the approach to meeting that challenge has been twofold. First, the College is refining its 
current assessment programs to focus on those elements which it has found to be most useful for its own purposes and also 
most common among its principal external requirements. Second, the College has begun to implement the programs in a process 
designed to emphasize the systematic documentation of important assessment results while also minimizing the effort needed 
to administer them. Sharpening the focus of the current assessment programs is described next, followed by description of the 
process in which the assessment programs are being administered. 

3 Basic elements of an outcomes-based assessment program 

While the outcomes assessment plans developed to date reflect Georgia Tech’s evolving philosophy of the self-evaluation of its 
educational programs, their development was also g uide d considerably by the exp ectati ons of external constituents, especially 
re-accreditation guidelines. For example, the undergraduate assessment plans in architecture, computing, and engineering 
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reflect the guidelines of their respective accreditation boards, whereas the undergraduate assessment plans in the sciences and 
liberal arts reflect more the guidelines from the Board of Regents. At the graduate level, all the Institute’s plans mirror a 
response to the requirements set forth by SACS, but they were also influenced by their respective undergraduate plans that 
were developed earlier. As such, the plans presently cover a multitude of elements related to assessment. For example, among 
the elements to be found in many plans are: summaries of unit strategic plans with their associated mission and vision 
statements; overviews of the unit administrative structures, and assessment responsibilities and procedures; descriptions of 
the unit strengths, challenges, faculty and facilities; tables of data presenting institutional support and resources; discussions 
of the program purpose and educational objectives; lists of expected student outcomes, program outcomes, and assessment 
methods; and examples of use of results. While these elements satisfied the particular reporting requirements for 
re-accreditation, relatively few of these elements are essential to the on-going assessment of a program’s performance. 

In general, the College of Engineering has found that four elements seem to be the most useful for its internal needs and also 
common to the requirements of its principal external constituents (Board of Regents, ABET, and SACS). The four elements are: 

1) a concise statement of the purpose of a degree program and its general educational objectives; 2) a list of the principal 
expected outcomes to be achieved by its graduates; 3) a list of the methods used to assess student achievement of the 
expected outcomes; and 4) a description of the process used to systematically document the use of assessment results. 

In large measure, these elements address the four basic questions that most accreditation visitors have in mind when reviewing 
a degree program, and certainly what faculty should have in mind when evaluating their curricula, namely: 1) What kind of 
career/lifetime preparation does the degree program seek to provide?; 2) What kinds of abilities are students expected to 
exhibit after graduating from the program?; 3) How does the program assess achievement of these skills and abilities by its 
graduates?; and 4) How does the program systematically evaluate and act upon the assessment results it collects? 

The College of Engineering has spent a considerable amount of effort developing and refining the first three elements [2]. The 
last element, however, is one that is receiving increased attention. While the college can readily provide examples of the use of 
assessment results, it has learned as a consequence of two re-accreditation reviews that demonstrating there is a systematic 
process in place to document and act upon assessment results is quite a different matter. Some observations about Georgia 
Tech’s experiences in addressing each of these four elements are the following. 

Developing a degree program’s purpose and general educational objectives was relatively straightforward. Because the Institute, 
College, and Schools had already developed strategic plans, mission and vision statements existed at each level. Although the 
strategic plans were developed with a much broader perspective in mind than specific degree programs, they often contained 
either explicit or implicit statements relevant to the degree programs. Thus, in many cases, units were able to readily develop 
brief statements describing the purpose of their degree programs and their general educational objectives by identifying or 
enhancing those statements within the strategic plans that related to the degree programs. Ideally, of course, assessment 
planning and strategic planning should be mutually supportive, that is, assessment results should inform strategic planning, and 
strategic planning should inform what should be assessed. In the first pass at developing assessment plans, strategic planning 
informed assessment planning, but not vice versa. However, the Institute is now updating its strategic plan from 1995, including 
those at the college and school level, and it is explicitly incorporating into the process mechanisms to link strategic planning with 
outcomes assessment. This will be an important step toward assimilating outcomes assessment into on-going Institute planning 
processes. 

The principal expected outcomes for graduates for each degree program were created pretty much from scratch. This element, 
unlike the other elements, was one in which the College (and Institute) had not addressed significantly in other related efforts, 
such as strategic planning, and it required considerable time and effort. To a large degree, developing written expected 
outcomes required the faculty to state, explicitly, what had been previously mostly their unstated expectations of their 
graduates. Faculty and students were involved in the process, as well as employers, others in industry, and alumni. The expected 
outcomes were, in large part, a natural extension of the program purpose and educational objectives. However, in some cases, 
during the process of developing the expected outcomes, it became clear that the program purpose and educational objectives 
needed some rethinking. As to be expected in attempting to do something this new, at all levels, and across the Institute, some 
expected outcomes were stated well and were quite specific, while others could still benefit from further refinement. 

Developing a list of the methods used to assess student achievement of the expected outcomes depended mostly on identifying 
the feedback mechanisms that were currently in use. The Institute, College, and many Schools had long been using surveys and 
interviews of alumni, students, and employers, monitoring the results from national examinations and competitions, and seeking 
feedback from various faculty, student, and industry groups. Thus, developing a list was mostly a matter of recognizing that 
many of the feedback mechanisms already in use were, in fact, methods of assessment, even if they had not otherwise been 
thought of or described in those terms before. A major challenge, however, was attempting to map meaningfully how the 
methods employed were believed to assess the outcomes. More often than not, the methods employed were indirect measures at 
best, and they often were measures of multiple outcomes. As such, the College approach has been to view the results of these 
methods more collectively than individually. That is, while an individual method may have its limitations, collectively, all the 
methods provide a reasonably informative picture. 

Finally, presenting documentation that assessment results are collected was fairly easy, e.g., benchmarking data, survey 
analyses, minutes of advisory board meetings; however, presenting documented evidence of the systematic use of assessment 
results was not an easy task. Rarely are important curriculum changes the result of a single survey, or one meeting of an 
advisory board, etc. They are typically the result of input from many sources, often oveLa. period of time, and frequently involve 
many faculty meetings and committees. As such, assembling materials in a manner so as to document the connection between a 
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given result and a specific curriculum change is very difficult. Historically, academia has not thought about documenting and 
archiving such materials in a manner suitable for accreditation or other external reporting requirements. Demonstrating that 
one has “closed the loop" between assessment results and curricular change is a worthy goal, but it certainly is not a 
straightforward process. As such, one area in which the College is devoting additional attention in its assessment plans is to 
include a section on the system of record keeping used to document the use of results. The intention is to bring more clarity as 
to the system of record keeping the units will employ to document regularly the use of assessment results. 



4 Sustaining an outcomes-based assessment program 

Now that the Institute has met the challenge of creating outcomes assessment programs for all its degrees, it now must meet 
the challenge of sustaining them as part its on-going operations. Initially, this was felt not to be a particularly significant 
challenge because most of the assessment methods used already had on-going data collection cycles, i.e., some data was 
collected quarterly, some annually, some bi- or tri-annually. It was believed that evaluation of the data collected would naturally 
"drive” the process of assessment. For purposes of the College’s self-evaluation, this was a reasonable belief and a reasonably 
effective process for quite some time. The College has long been collecting data, seeking feedback, and acting on the results to 
improve its curriculum. However, what the College has not been doing is documenting systematically how the data collected 
connects to program improvements. In essence, for purposes of demonstrating adequate self-regulation, external constituents 
required more archival evidence than the College had been accustomed to collecting. The challenge, then, has been to create a 
process that meets these external expectations, provides useful information for self-evaluation, and is not a significant time 
consuming task. 

To this end, the Institute is now implementing a process that is comprised of two elements. The process requires that each 
degree program: 1) prepare a brief annual report summarizing its major assessment activities each year; and 2) conduct a 
review, and prepare a brief report, of its overall performance every five years. 

The first round of annual reports is due in the fall of 1999. The purpose of the reports is to provide a mechanism to document, 
and archive, on a regular basis the use of assessment results. Each academic unit will prepare a summary report highlighting the 
assessment efforts of its degree programs for the past year. By keeping the reports brief, it is intended to keep them: focused 
on the major areas for improvement; suitably informative for dissemination across the Institute; and more likely to be acted 
upon. Over time these reports will form the basis of a system of record keeping that is intended to document the systematic 
use of assessment results. 

The annual reports will highlight three areas: significant modifications to the program assessment plans, if any; noteworthy 
results of assessment activities and their dissemination (within and outside the degree program); and major actions proposed or 
taken as a consequence of the evaluation of the assessment results. An annual reporting cycle was selected primarily because 
much of what occurs in academia occurs around the academic year. Additionally, more frequent reporting was viewed as too 
burdensome, and less frequent reporting would not likely be viewed by external constituents as a sufficiently "on-going" and 
"continuous" process. 

The first round of five-year program reviews for the Institute is scheduled for AY 1999-2000 (although the College of 
Engineering actually began its cycle in the spring of 1999). The purpose of the reviews is very similar to the annual reports 
except that they are intended to provide an opportunity for longitudinal evaluation of assessment results and to conduct a 
periodic evaluation of the effectiveness of the assessment process itself. Within the College of Engineering, these program 
reviews will become part of the School-wide reviews that are now conducted on a similar cycle. The School-wide reviews are 
conducted with the assistance of an external committee of visitors that interacts with faculty, students, staff, and 
administrators, mostly within the School, and provides the School, College, and Institute with their thoughts and comments on 
the School's overall directions, operations, and performance in all areas. Although evaluation of the School's degree programs 
has always been a part of the review, it has not previously been conducted from the perspective of outcomes assessment. 

Future School reviews will include within them a review of the performance of the School’s degree programs, as has been done 
before, but it will also include an evaluation of the School’s assessment plans and its implementation of them. 

The program reviews will be phased in so that ultimately approximately one-fifth of the Institute degree programs are reviewed 
each year. All degree programs will have completed their first five-year review by 2003, in time for the next general ABET 
review for engineering that occurs that year as well as for the next general SACS review for all programs scheduled for 2004. 

5 Suggestions for developing an outcomes-based assessment program 

As mentioned before, as a result of either the policies of the Board of Regents, ABET criteria, or SACS criteria, all of the 
Institute's undergraduate programs and graduate degrees now have outcomes assessment programs. Much of what was learned 
in responding to the Board of Regents mandate in 1992 was used to prepare for EC 2000 in 1997. Similarly, a great deal of the 
lessons learned through EC 2000 were subsequently applied in preparation for the SACS re-accreditation in 1998. As a result of 
these experiences, seven suggestions are offered for. those developing outcomes-based assessment programs. 

1. Focus on what is important to your institution first; focus on what is important to external 
constituents second. 



One of the first questions faculty frequently ask, especially if the assessment program is being developed in response to 
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external constituents, is: "What do they want?" "They" could be a governing board, accreditation body, or other external 
constituent. The focus should be on what the college wants and needs. Assuring that the assessment programs developed align 
with other external requirements should be a secondary issue, albeit an important one. Getting assistance and buy-in from 
faculty, administrators, and others, is easier if the effort is focused on providing useful information for the college. 
Fundamentally, most external constituents, such as ABET, require the development of educational objectives and program 
outcomes consistent with the mission of the college, an assessment process that demonstrates they are being achieved, and a 
system of continuous evaluation and improvement [1]. There is ample room within that structure to both fulfill institutional 
needs and meet external requirements. 

2 . Improve existing assessment measures and processes first. 

Most, if not all, engineering colleges have been doing continuous improvement and outcomes assessment for a long time. They 
may not have thought of it in those terms, but thoughtful evaluation and diligent monitoring of the effectiveness of the 
curriculum have always been major activities of most any engineering college. Georgia Tech had been conducting alumni surveys, 
holding external advisory board meetings, benchmarking against peer institutions, etc, for quite some time. It may not have been 
as systematic as it could have been, and the quality of the effort in some areas may have needed some attention, but the 
quantity of effort was considerable. Indeed, there seems to be a predisposition among some faculty to assume that because 
external constituents are now proposing requirements that appear to be a dramatically new and different, such as EC 2000, then 
they must propose in response dramatically new and different approaches of evaluation. Such is not the case. Georgia Tech 
made considerable strides in developing its assessment programs by focusing on improving existing practices. It also helped in 
getting support from the faculty to develop more systematic assessment programs by working on improving existing activities 
that were already viewed as useful and helpful to their operations. 

3. Share information and collaborate as much as possible. 

Raising the quality of assessment activities improves significantly when faculty share their efforts with other academic units. 
Further, for those units that have not yet given outcomes assessment much thought, collaboration greatly facilitates their 
movement up the learning curve by having the opportunity to benefit from the lessons learned from those units that have given 
assessment some thought. Capitalizing on each other's best ideas saves time and improves the results. For example, alumni 
surveys in the past had been done by individual schools and the results were seldom shared with other schools. Much of what 
constitutes an engineering education is often common among the different disciplines. By developing a College-wide survey, the 
breadth and quality of the alumni surveys was improved, the disciplinary depth was still maintained, the support of the alumni 
office was used more efficiently, and a more thorough analysis was performed involving a larger data set of responses. 

4. Clarify terminology and establish the key elements of the assessment plans early in the 
development process. 

Mission, vision, objectives, goals, strategies, tactics, outcomes, methods, processes, programs, etc, etc. The field of 
assessment involves a significant amount of terminology. It is helpful to reach an agreement early in the development process as 
to what is meant by the terms that are used. It is important not to get bogged down in debates such as whether "goals" are the 
means to achieve "objectives" or vice versa. It is more important to simply agree on what is meant by whatever terms are used. 

It is also helpful to agree early in the development process on what will be the key elements required of each assessment plan. A 
common framework of key elements helps to identify opportunities for college-level or university-level collaboration, assures 
that important components of outcomes-based assessment are addressed in every degree program, and reduces faculty 
frustrations in working in this relatively new arena by keeping the program development focused. Within the overall framework, 
individual units have ample opportunity to develop assessment plans that reflect their disciplinary and/or institutional 
uniqueness. 

5. Identify benchmark institutions and key constituents. 

Without knowing one’s academic competition, it is difficult to know if the college is improving, holding steady, or losing ground. 
External calibration with respect to institutions with which a college aspires to be more competitive can be very enlightening. 
Using national averages can be helpful, but it probably will not provide as sharp a picture. Similarly, regular feedback from the 
key stakeholders helps identify the most important opportunities for improvement. Obtaining data is a major part of 
assessment, and there is a lot of data that could be gathered. Having defined benchmark institutions and identified key 
stakeholders facilitates keeping the data gathering activities focused and meaningful. 

6. Gather data, and lots of it. 

Most any piece of data has its limitations. Indeed, some data may only remotely measure what the college seeks to evaluate. 

Many aspects of education are not easily measured. Neither of these issues, however, should become roadblocks to gathering 
data. Many academic processes involve measures that are limited in their scope or are attempting to evaluate very subjective 
issues. Nonetheless, evaluating lots of data, especially over time, and from a variety of sources, eventually provides an 
informative picture. 

7. Develop a system to document the use of results. _ __ __ 
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This is the one area that represents the greatest challenge for most academic institutions, and certainly for U.S. colleges of 
engineering implementing EC 2000. For administrative-type actions, this tends not to be a major issue because of the "paper 
trail" nature of most administrative processes. For curriculum-type actions, however, the tendency is to simply make the change 
and to not necessarily document the deliberations and inputs leading to the decision to make the change. As a consequence of 
the SACS re-accreditation visit, the Institute assessment plans are being modified to have a section focused on the record 
keeping system used to document the use of results, in addition to the records of evidence of the use of results. 

6 Conclusion 

Georgia Tech has learned many lessons in developing its outcomes assessment programs [2][3]. Its most recent lesson learned 
is the need to refine and consolidate its efforts to assure that outcomes assessment is sustainable within the Institute's limited 
resources while also fulfilling the Institute's internal and external needs. The challenge is considerable. In a recent report to 
SACS, Georgia Tech estimated, conservatively, that it will spend nearly $750,000 next year on outcomes assessment activities. 

The success of outcomes assessment on most campuses will depend in part on the degree to which institutions are able to 
satisfy the needs of multiple constituents without the need for multiple assessment programs. For the most part, engineering 
education has long been practicing outcomes assessment. It may not have thought of it in those terms, and it may not have 
documented well its efforts, but self-evaluation and program improvement have been major activities in most colleges of 
engineering. What has changed is the need to share those efforts and results with a diverse array of external constituents. 
Within the College of Engineering at Georgia Tech, the principal strategy has been to develop outcomes assessment plans around 
those elements that are most useful internally and also common to external requirements, and to implement them in a 
sustainable process that provides adequate documentation. 
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